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Abstract 

Student problem behaviors in early elementary school are associated with fewer learning opportunities, poor relationships 
with teachers, and academic and behavioral problems, including increased risk of emotional/behavioral disorders (EBDs). 
This study examined the effect of BEST in CLASS—Elementary (BEST in CLASS-E), a Tier 2 intervention delivered by 
teachers, on student problem behavior, academic achievement, teacher—student relationships, and classroom quality using a 
randomized controlled trial design. Participants were 45 kindergarten to grade 3 students identified at risk of development 
of EBDs and their 26 teachers from three elementary schools located in an urban school district. BEST in CLASS-E was 
found to decrease teacher-reported student problem behavior and increase teacher—student closeness; no effects were 
noted for academic achievement or overall classroom quality. Results suggest the promise of BEST in CLASS-E as a Tier 2 


intervention delivered by teachers in elementary classrooms. Implications and limitations of the study are discussed. 
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Approximately 0.5% to 1% of the school-age population 
receives special education services for an emotional/behav- 
ioral disorder (EBD) under the Individual with Disabilities 
Education Act (IDEA; Forness, Freeman, et al., 2012). Yet, 
epidemiological studies indicate that many more children 
and youth are at risk of or have an EBD with prevalence 
rates ranging anywhere from 11.5% to 30% (e.g., Forness, 
Kim, et al., 2012; Ringeisen et al., 2017). Whether or not 
students are formally identified and found eligible under 
IDEA, they attend school and their chronic problem behav- 
iors can have a negative impact on their long-term academic 
and behavioral success in school (Kellam et al., 1998; 
Myers & Pianta, 2008; Spilt, Koomen, et al., 2012). One 
such outcome identified in the literature is the development 
of negative and coercive interactions with their teachers and 
peers (e.g., Doumen et al., 2008). 

If chronic problem behaviors are not addressed early on, 
students with and at risk of EBD may develop negative 
interaction patterns with adults in their lives, including their 
teachers (see Gunter & Coutinho, 1997; Patterson, 2002, for 
a discussion). When students and teachers engage in negative 
interactions, teachers often lack the training and skills to 
proactively address these interactions (O’Conner et al., 2011; 


Sutherland et al., 2008). In fact, Reinke and colleagues (2011) 
reported that only 17% of elementary school teachers rated 
themselves as having training in practices to address chronic 
problem behaviors in their classrooms. Rather than using 
evidence-based practices that de-escalate negative interac- 
tions and teach students skills to engage positively, teachers 
often respond with harsh, negative feedback (McClowry 
et al., 2013). To escape these negative interaction patterns, 
teachers may also decrease students’ learning opportunities 
by providing fewer opportunities to respond and less atten- 
tion (Farmer et al., 2016; Sutherland et al., 2008). The cumu- 
lative negative outcomes associated with sustained negative 
interactions and lack of learning opportunities seriously 
affect students’ future academic, social, and behavioral out- 
comes (Brock et al., 2008). Therefore, a critical need exists to 
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address chronic problem behavior and negative teacher— 
student interaction patterns. 


Tier 2 Interventions Targeting 
Students With or at Risk of EBD 


Over the past 30 years, there has been an increasing empha- 
sis on the use of positive behavioral intervention and sup- 
ports (PBIS) to address problem behaviors in schools. The 
focus of PBIS is to provide tiered systems of supports early 
and systematically with an emphasis on prevention and 
amelioration of problem behaviors (Sugai & Horner, 2009). 
Within a PBIS framework, Tier 2 interventions provide 
additional supports to students whose behavior continues to 
be problematic after receiving high-quality Tier 1 supports. 
Typically provided to a smaller group of students in need of 
supports beyond those of Tier | or more universal strate- 
gies, a number of Tier 2 interventions have been developed 
and have effectively addressed the behavioral needs of 
many students in school settings (for a review, see Anderson 
& Borgmeier, 2010; Bruhn et al., 2014). BEST in CLASS— 
Elementary (BEST in CLASS-E; Sutherland et al., 2019) is 
one such Tier 2 intervention designed to address the needs 
of early elementary-aged students who have been identified 
as at risk of EBD and engage in chronic problem behaviors 
and negative interactions with their teachers. 


Overview of BEST in CLASS-E 


Through funding from the U.S. Department of Education, 
Institute of Education Sciences, BEST in CLASS-E was 
designed to increase positive teacher—student interactions 
and increase learning opportunities for students in kinder- 
garten to grade 3 who are at risk of or with EBD. Adapted 
from the original BEST in CLASS for delivery in early 
childhood classrooms (see Conroy et al., 2018), BEST in 
CLASS-E was founded on Sameroff’s (2009) transactional 
theory, which when applied to interactions between stu- 
dents and teachers postulates that student behavior (whether 
positive or negative) influences the behavior of teachers 
and vice versa. These transactions between teacher and stu- 
dent influence, and are influenced by, the broader ecology 
within the classroom (Bronfenbrenner, 1979). An iterative 
development process with teachers and families of young 
students with and at risk of EBD, including pilot testing in 
urban elementary schools, indicated both promise for the 
intervention and contextual fit in the schools included in the 
current study (see Sutherland et al., 2019, for a description 
of this intervention development process). 

BEST in CLASS-E is comprised of three components: 
(a) a teacher training workshop on the BEST in CLASS-E 
practices; (b) a teacher resource manual, which provides 
additional materials and supports; and (c) 14 weeks of 
practice-based coaching. Through these three components, 


teachers learn how to use five instructional practices (i.e., 
supportive relationships, rules, precorrection, opportunities 
to respond, praise) that, when used effectively, increase 
learning opportunities and positive interactions with stu- 
dents with and at risk of EBD in their classroom who engage 
in elevated rates of chronic problem behaviors. In addition, 
teachers learn proactive ways to engage the students’ fami- 
lies in their child’s education (for a description, see Conroy, 
McKnight, et al., 2019). 

A unique aspect of BEST in CLASS-E in comparison 
with other Tier 2 interventions, which are implemented in 
small groups or individually, is that teachers learn to embed 
the BEST in CLASS-E practices within naturally occurring 
learning opportunities as they interact with these students 
throughout their school day. BEST in CLASS-E is consid- 
ered a “value-added” intervention, in that teachers may 
already be using many of the BEST in CLASS-E instruc- 
tional practices during learning activities in their class- 
rooms. When implementing BEST in CLASS-E as designed, 
teachers are taught to increase the quantity and quality of 
delivery of the BEST in CLASS-E instructional practices 
with focal students who are engaging in sustained rates of 
problem behaviors. For example, most teachers have class- 
room rules to teach all students in the classroom the behav- 
ioral expectations (i.e., a Tier 1 intervention). However, 
BEST in CLASS-E provides teachers with instruction and 
support to help them learn how to teach focal students to 
learn the classroom rules, and embed the use of rules more 
consistently and frequently within each instructional activ- 
ity in the classroom, increasing the intensity and dosage of 
intervention. Likewise, teachers commonly provide stu- 
dents opportunities to respond during instructional activi- 
ties; however, BEST in CLASS-E helps teachers learn how 
to increase the frequency and quality of opportunities to 
respond with the focal students to increase their engage- 
ment in the instructional activity. As a result, students 
engage in fewer problem behaviors and have increased 
learning opportunities resulting in fewer negative interac- 
tions with their teachers. 


Overview of Current Study 


Previous research on BEST in CLASS has demonstrated 
positive outcomes for teachers (Conroy, Sutherland, et al., 
2019) and children (Conroy et al., 2018; Sutherland, 
Conroy, Algina, et al., 2018) in early childhood settings. To 
illustrate, Conroy, Sutherland, et al. (2019) found that teach- 
ers in the BEST in CLASS condition had increased self- 
efficacy (ES range = 0.50—0.78) and Classroom Assessment 
Scoring System (CLASS; Pianta et al., 2008) domain scores 
(ES range = 0.47-0.65). In terms of child outcomes, 
Sutherland, Conroy, Algina, et al. (2018) found reductions 
in teacher-reported problem behavior (ES = —42) on the 
Social Skills Improvement System—Rating Scale (SSIS-RS; 
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Gresham & Elliott, 2008), as well as improved closeness 
(ES = 0.26) and reduced conflict (ES = —0.29) on the 
Student—Teacher Relationship Scale (STRS; Pianta, 1993). 
Although positive outcomes were found in early childhood 
settings, an investigation of the initial efficacy of BEST in 
CLASS-E for elementary students with and at risk of EBD 
who engage in chronic problem behaviors has not been con- 
ducted. This article reports findings from an initial investi- 
gation examining the effects of BEST in CLASS-E in 
elementary school classrooms. Several social, emotional, 
and behavioral outcomes for students who are at risk of or 
with EBD were examined, as well as the overall classroom 
quality. The following research questions were addressed: 


Research Question 1: What is the effect of BEST in 
CLASS-E on student problem behavior? 

Research Question 2: What is the effect of BEST in 
CLASS-E on student academic achievement? 

Research Question 3: What is the effect of BEST in 
CLASS-E on student-teacher relationships? 

Research Question 4: What is the effect of BEST in 
CLASS-E on classroom quality? 


Method 


Setting and Participants 


The researchers recruited teacher participants from three 
elementary schools in an urban school district in a mid- 
Atlantic state. The participating schools served predomi- 
nantly African American students (94%, 93%, and 98%, 
respectively) from a low-income community (82%, 96%, 
and 96% free and reduced lunch, respectively). The mean 
number of students per school was 364 (SD = 73.9). 

The schools indicated that no formal schoolwide positive 
behavior interventions and supports (SWPBIS) were in 
place, although evidence of informal application of SWPBIS 
were observed. Tier | supports were implemented with some 
variability across classrooms, and teachers engaged in class- 
room management practices, but at low levels of fidelity. To 
illustrate, the average CLASS (Pianta et al., 2008) scores for 
BEST in CLASS-E and business as usual (BAU) condition, 
respectively, for the subscales of Classroom Organization 
(3.91 and 3.93), Instructional Support (1.95 and 1.82), and 
Emotional Support (4.62 and 4.24) at pretest in the current 
study were considerably lower than scores found in other 
studies. For example, Jennings et al. (2017) noted CLASS 
scores of 4.86 (Classroom Organization), 2.75 (Instructional 
Support), and 4.92 (Emotional Support) in 224 elementary 
classrooms, whereas Sandilos et al. (2014) noted scores of 
5.16 (Classroom Organization), 3.03 (Instructional Support), 
and 5.26 (Emotional Support) in 426 elementary classrooms. 
CLASS scores in the current study would suggest a lack of 
Tier | supports at the classroom level. 


Teachers. A total of 26 teachers participated in the study, 
with 14 in the BEST in CLASS-E intervention group and 12 
in BAU (see Figure 1). Teachers were eligible for inclusion 
if they met the following criteria: (a) taught in kindergarten 
to grade 3, (b) served at least one child identified as being at 
risk of EBDs, and (c) consented to participate. There were 
no significant differences between the BEST in CLASS-E 
intervention group and the comparison group on the teacher 
demographic variables in Table 1. 


Students. A total of 45 students participated in the study, 
with 25 in the BEST in CLASS-E intervention group and 20 
in the BAU group (see Figure 1). Students who met the fol- 
lowing criteria were eligible for participation: (a) enrolled 
in a participating teacher’s classroom, (b) presence of exter- 
nalizing behaviors that interfere with participation in the 
classroom (e.g., disruption, aggression) as indicated by the 
Systematic Screening for Behavior Disorders (SSBD; 
Walker et al., 2014), and (c) parental/guardian consent to 
participate. To determine eligibility for participation, in 
Stage 1, teachers nominated up to five students in their 
classroom who engaged in chronic problem behavior based 
on a list of example and nonexample behaviors. Caregiver 
consent was then obtained and systematic screening for risk 
of EBDs took place using the SSBD Stage 2. After screen- 
ing, one to two students per classroom were selected to par- 
ticipate in the study, depending upon returned caregiver 
consents and the most elevated scores on the SSBD. All 
students who screened into the study met the criteria for “at 
risk” as defined by the normed cutoff criteria on the SSBD 
(see Walker et al., 2014). There were no significant differ- 
ences between the intervention and comparison groups on 
the student demographic variables in Table 1. 


Measures 


SSBD. The SSBD (Walker et al., 2014) is a three-stage mul- 
tigate screening system designed to proactively identify stu- 
dents who are at risk of negative developmental outcomes 
associated with their internalizing and externalizing behav- 
ior patterns. The tool combines teacher ratings of the fre- 
quency and intensity of student adjustment problems in 
school with trained observer ratings of student’s adaptive 
and maladaptive behaviors. In the present study, we used 
the externalizing subscale in which students are categorized 
into risk levels associated with four scales: critical events, 
aggressive behavior, maladaptive behavior, and adaptive 
behavior. The SSBD suggests that the first two stages be 
completed, and allow the third stage, observation by trained 
observers, to be optional; we used the first two stages to 
identify focal students. In Stage 1, teachers rank the top five 
students who engage in externalizing behavior based on a 
list of example and nonexample behaviors. In Stage 2, stu- 
dents with caregiver consent are then rated by their teacher 
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Consented Teachers = 33 
Consented Students = 71 


Teacher and student attrition prior to baseline 
Teacher Attrition: n = 7 
1 left teaching 
2 declined after initial consent 
1 did not meet eligibility criteria 
1 took extended leave 
2 no qualifying children based on consent or screening 


Student Attrition: n = 26 
22 did not meet screening criteria 
4 were not eligible (e.g., teacher left, child moved, parent 
consent form returned too late) 


Randomization of teachers 
n = 26 teachers 
n= 45 Students 


BEST in CLASS—Elementary Business as Usual (Comparison) 
Condition at Baseline: Condition at Baseline: 
Teachers n= 14 Teachers n= 12 
Students n = 25 Students n = 20 


Teacher and student attrition after baseline 
Teacher Attrition: n= 2 

1 left for extended leave 

1 left after student left schools 


Student Attrition: n = 3 
2 students left due to teacher taking extended 
leave 
1 student left school 


BEST in CLASS—Elementary Business as Usual 
Condition at Post-Test: Condition at Post-Test: 
Teachers n = 13 Teachers n=11 
Students n = 24 Students n= 18 


Figure |. The CONSORT flow diagram for teacher and student participation in the study. 
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Table |. Participant Demographics by Study Group. 


Participant BAU BEST in CLASS-E Total 
Teachers n= 12 n= 14 26 
Age range (years) 
18-25 | 3 4 
26-35 5 6 11 
36-45 3 2 5 
46-55 3 2 5 
Above 55 0 I | 
Gender 
Female 12 14 26 
Hispanic/Latinx 0 | | 
Other 0 I | 
Non-Hispanic/Latinx 12 13 25 
African American/Black 7 4 11 
Caucasian/White 5 8 13 
Other 0 | I 
Years teaching M = 6.33, SD = 6.95 M = 6.07, SD = 7.85 M = 6.19, SD = 7.29 
Education 
Bachelor’s degree 8 6 14 
Master’s degree 4 8 12 
Grade taught 
Kindergarten 2 4 6 
First 3 4 7 
Second 3 2 5 
Third 2 2 4 
Special education 2 2 4 
Students n= 20 n=25 n=45 
Age M = 7.48, SD = 1.35 M = 7.35, SD = 1.31 M = 7.39, SD = 1.31 
Gender 
Male 17 20 37 
Female 3 5 8 
Non-Hispanic/Latinx 16 25 4l 
African American/Black | 0 | 
Caucasian/White | 0 | 
Other | 0 | 
Grade 
Kindergarten 4 v4 11 
First 5 7 12 
Second 5 6 II 
Third 4 5 9 
Special education 2 0 2 


Note. BAU = business as usual condition; BEST in CLASS—E = treatment condition; CLASS = Classroom Assessment Scoring System. 


on the critical events index (CEI) and combined frequency 
index (CFI). Both stages exhibit strong psychometric prop- 
erties. Stage | shows acceptable test-retest stability esti- 
mates (e.g., 69% of students ranked as the top three 
externalizers were also ranked in the top three at a ranking 
Time 2, and Spearman’s rank order coefficients between 
Stage | rankings at two time points reveal a mean r coeffi- 
cient of .76 for externalizers; Walker et al., 2014). Stage 2 
also demonstrates internal consistency, with Adaptive 


Behavior Scale alphas of .85 and .88, across the two ratings 
1 month apart. For the Maladaptive Behavior Scale, these 
coefficients were .82 and .87 (Walker et al., 2014). 


SSIS-RS. The SSIS-RS (Gresham & Elliott, 2008) is a 
76-item teacher-report measure, allowing for the evaluation 
of social skills and problem behaviors of young students. 
Each item on the SSIS-RS is rated on a 4-point frequency 
scale, with responses ranging from 0 (never) to 3 (almost 
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always). Items are grouped into two subscales: Social Skills 
(e.g., completes tasks without bothering others) and Prob- 
lem Behaviors (e.g., talks back to adults), with higher scores 
indicating more social skills or more problem behavior. For 
the current sample, internal consistency was acceptable 
with Cronbach’s alpha equal to .93 and .95 for Social Skills 
at pretest and posttest, respectively, and .89 and .95 for 
Problem Behavior at pretest and posttest, respectively. 


Woodcock-Johnson-IIl_ Brief Battery (W)-III). Academic 
achievement was measured using two subtests of the WJ- 
Ill (Letter-Word Identification and Math Applied Prob- 
lems; Woodcock et al., 2007), which are ideal for measuring 
academic progress multiple times in | year. The Brief 
Reading and Brief Math cluster raw subscales were used 
(Woodcock et al., 2007), higher scores indicating more 
skills. The subscales of the WJ-III consistently demon- 
strate high reliability with internal consistency of .80 or 
higher. 


STRS. Designed to measure teachers’ perceptions of their 
relationships with students, the STRS (Pianta, 1993) is a 
teacher-report measure with subscales assessing domains of 
Closeness (the degree of warmth, positive emotions, and 
open communication between teacher and student) and Con- 
flict (the degree of negative interactions and emotions involv- 
ing teacher and student). Items are rated on a 5-point 
Likert-type scale, with responses ranging from | = definitely 
does not apply to 5 = definitely applies and scores summed 
for each domain, with higher scores indicating more conflict 
or more closeness. For the current sample, internal consis- 
tency was acceptable with Cronbach’s alpha equal to .83 and 
.78 for Closeness at pre- and posttest, respectively, and .80 
and .84 for Conflict at pre- and posttest, respectively. 


Problems Preparing Children for Academic Success (PPCAS). 
Teachers completed the PPCAS scale, which was adapted 
from the School Staffing Survey (National Center for Educa- 
tion Statistics, 1993). Teachers report on the following prompt, 
“How much of a problem are the factors below in preparing 
your children to success academically?” Seventeen factors are 
listed and include home/family life, parent cooperation/sup- 
port, child health, inadequate nutrition, low intelligence, cul- 
tural differences, English proficiency, nonstandard English, 
special learning problems, behavioral problems (disruptive), 
inadequate supplies, students not ready academically, students 
have attention problems, and student tardiness/absenteeism. 
Teachers respond using a 4-point scale from 1 = not a prob- 
lem to 4 = serious problem. Ratings were averaged across the 
17 items (a = .83), with higher scores indicating higher 
teacher-perceived classroom-level adversity. No differences 
were present between BEST in CLASS-E and BAU class- 
rooms on classroom-level adversity (/ = 3.00 and 3.01, SD 
= 0.61 and 0.64, respectively). 


CLASS. The CLASS (Pianta et al., 2008) is an observational 
instrument designed to evaluate classroom quality. The 
CLASS addresses 10 separate dimensions of classroom 
experiences over three broad domains, including Emotional 
Support, Classroom Organization, and Instructional Support. 
Trained observers scored each of the dimensions supporting 
the broad domain using a | to 7 scale with three categories: 
low (1, 2), mid (3, 4, 5), and high (6, 7). A minimum of four 
observation cycles were completed on each classroom, and 
dimension scores were averaged across cycles to create broad 
domain scores. A higher score indicates high quality for each 
dimension. For the current sample, internal consistency was 
acceptable with Cronbach’s alpha equal to .81 for Emotional 
Support at pretest and .81 for posttest, .93 for Classroom 
Organization at pretest and .93 for posttest, and .88 for 
Instructional Support at pretest and .85 for posttest. 
Certified observers conducted observations using the 
CLASS at pretest and posttest. Observers participated in a 
2-day training led by a certified CLASS trainer and passed 
the reliability test to achieve initial certification. All observ- 
ers recalibrated prior to posttest data collection. Interobserver 
agreement data were collected on 23% of all CLASS obser- 
vations, and the mean intraclass correlations (ICCs) were 
81, .87, and .58 for Emotional Support, Classroom 
Organization, and Instructional Support, respectively. 


Treatment integrity measure. The Treatment Integrity Instru- 
ment for Elementary School Classrooms (TIES; Sutherland 
et al., 2017) is an observational measure in which raters 
assess teachers’ extensiveness (i.e., adherence; five items), 
quality of delivery (i.e., competence; five items), and stu- 
dent responsiveness (one item) using a 7-point Likert-type 
scale. Adherence ratings were comprised of two key com- 
ponents: thoroughness and frequency. Anchors on the 
adherence and student responsiveness items range from not 
at all to very extensive. Competence ratings were comprised 
of estimates of the skillfulness of delivery, timing, and 
responsiveness to a given child and situation. Anchors on 
the competence items range from very poor to excellent. 
Trained observers conducted 87 observations in BEST in 
CLASS-E and BAU classrooms at pretest or posttest; a sec- 
ondary observer conducted reliability checks during 34 
observations (39.1% of the total observations). ICCs were 
computed for each item on each scale. Cicchetti (1994) 
indicated that ICCs less than .40 reflect “poor” agreement, 
ICCs from .40 to .59 represent “fair” agreement, ICCs from 
.60 to .74 represent “good” agreement, and ICCs of .75 and 
higher represent “excellent” agreement. The mean ICC for 
the adherence scale was .82 (SD = 0.10, range = .68-.92), 
with all items reflecting “good” to “excellent” agreement. 
The mean ICC for the competence scale was .61 (SD = 
0.11, range = .52-.77), with items representing “fair” to 
“excellent” agreement. The ICC for the student responsive- 
ness item was .68, representing “good” agreement. 
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Design and Experimental Procedures 


This study was a randomized trial. Students were nested in 
teachers’ classrooms, and teachers were randomly assigned 
to the BEST in CLASS-E or to a BAU comparison condi- 
tion. Teachers were randomly assigned to a condition from 
within their school and when possible, grade level (i.e., kin- 
dergarten, grades 1, 2, 3). For example, in schools with two 
consented teachers per grade level, one teacher was ran- 
domly assigned to BEST in CLASS-E and one to BAU. To 
minimize contamination across conditions, teachers were 
provided information about their roles in the study and the 
importance of not discussing participation with any other 
teachers in their school. 

All study activities were approved by a human partici- 
pants protection board. Obtaining teacher consent and 
screening of student participants began approximately | 
month after the beginning of school to allow teachers to 
familiarize themselves with the students in their classrooms. 
Once teacher consent was obtained, students were screened 
to determine eligibility for participation and caregiver con- 
sent was received. Subsequently, pretest measures were 
completed. Following completion of pretest measures, ran- 
domization occurred (see Figure | for study flow). Teachers 
in the BEST in CLASS-E condition received the BEST in 
CLASS-E 1-day teacher training, and the following week 
began receiving weekly practice-based coaching (see 
“Method” section). In April, posttest measures were com- 
pleted with all participating teachers and students. 


Treatment and Comparison Conditions 


BEST in CLASS-E. Students in the BEST in CLASS-E condi- 
tion were exposed to BAU with the addition of BEST in 
CLASS-E. BEST in CLASS-E is considered a Tier 2 inter- 
vention, because teachers systematically identify specific 
focal students in their classrooms who are at risk of EBDs, 
and after receiving training and practice-based coaching to 
increase the quality and quantity of targeted instructional 
practices implement these practices with these students dur- 
ing instructional activities throughout the school day. The 
intention of BEST in CLASS-E is to increase a teacher’s 
use of specific instructional practices with focal students 
that facilitate positive teacher—student interactions and stu- 
dent engagement while decreasing the occurrence of stu- 
dent problem behaviors (e.g., disruption, defiance). This is 
accomplished through the implementation of three key 
components delivered to teachers: (a) the BEST in CLASS- 
E teacher manual, (b) the BEST in CLASS-E teacher 
workshop (a | -day didactic teacher training), and (c) practice- 
based coaching (14 weeks of one-on-one practice-based 
coaching with performance feedback). 

The BEST in CLASS-E teacher manual provides an over- 
view of the intervention components and implementation 


process. The manual is comprised of seven modules; five of 
the modules provide instructional support on the BEST in 
CLASS-E practices (i.e., supportive relationships, rules, pre- 
correction, opportunities to respond, and praise) and the final 
module helps the teacher learn to link these practices together. 
The first module during coaching focuses on supporting the 
teacher to partner with families of focal students, and this 
home-school partnership component is reviewed with teach- 
ers weekly. Each module includes a definition of each prac- 
tice and steps for successful implementation with focal 
students. Supporting research and high-quality examples of 
each practice are provided. Teachers are introduced to the 
manual at the BEST in CLASS-E teacher workshop, but con- 
tinue to use the manual throughout the 14 weeks of practice- 
based coaching as they master each module. 

At the BEST in CLASS-E teacher workshop, led by uni- 
versity faculty and coaches, teachers receive detailed infor- 
mation about the use of each BEST in CLASS-E practice 
with focal students. The training includes modeling, video 
exemplars, and hands-on activities that are used to facilitate 
discussion between teachers and coaches about the specific 
problem behaviors of focal students in their classroom. 

The BEST in CLASS-E coaching process begins the 
week after the teacher workshop and lasts for 14 weeks. The 
coaching process is cyclical, and teachers and coaches col- 
laboratively set a new goal each week for using a BEST in 
CLASS-E practice. Coaches and teachers spend 2 weeks 
focusing their goals on the implementation of each practice 
as well as how that practice can be linked with other BEST 
in CLASS-E practices in an efficient manner. Ultimately, 
the coaching is intended to extend what teachers have 
learned from the manual and workshop to facilitate their use 
of the BEST in CLASS-E practices with the identified focal 
students in their classroom. 

At each coaching meeting, a coaching plan is developed 
and includes specific ways the teacher will implement prac- 
tices to reduce focal student’s problem behaviors and increase 
engagement as well as how the coach will support the teach- 
er’s efforts. In addition to the development of a coaching plan, 
the coach also observes the teacher’s implementation of the 
plan in the classroom with focal students during the selected 
activity. The coach’s observation lasts approximately 15 min 
per teacher-student dyad during which time the coach 
observes and videotapes each teacher—-student dyad, takes 
anecdotal notes, and collects observational data on teachers’ 
use of the BEST in CLASS-E practices, teacher—student inter- 
actions, and student engagement and problem behaviors using 
the TIES (Sutherland et al., 2017). Following the teacher’s 
implementation of the plan and the coach’s observation, the 
teacher reflects and evaluates on the implementation of the 
plan and the coach summarizes the observation. The teacher’s 
reflection is discussed at a subsequent coaching meeting, 
while the coach provides detailed performance feedback data, 
video examples, and anecdotal notes. Following the reflective 
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and performance—feedback component of the meeting, the 
coaching cycle begins again with goal setting and the devel- 
opment of an action plan. See Sutherland et al. (2015) and 
Sutherland, Conroy, Algina, et al. (2018) for a more detailed 
description of the practice-based coaching component of 
BEST in CLASS-E. 


Coaches. Four coaches (75% female; 75% Caucasian, 25% 
African American) were part of BEST in CLASS-E. Coaches 
ranged in age from 30 to 36 years. All coaches held at least a 
bachelor’s degree, with 50% holding a master’s degree or 
higher, and 75% currently enrolled in a graduate program at 
the time of coaching. All coaches reported at least 1 year of 
teaching experience and 75% reported holding a teaching 
certificate. In addition, 50% of coaches reported at least 1 
year of practice-based coaching experience. 


BAU. Students in the comparison group experienced a BAU 
condition only. The BAU condition consisted of daily 
instructional activities typically offered by elementary 
teachers in classrooms. For the most part, classroom activi- 
ties began with a morning meeting, which was followed by 
a combination of small and large-group teacher-directed 
activities (e.g., early literacy, mathematics). Teachers in 
both BAU and BEST in CLASS-E conditions received the 
same professional development opportunities (other than 
BEST in CLASS-E teachers receiving BEST in CLASS-E 
training and coaching), and these trainings varied some- 
what across schools. In general, teachers received trainings 
on topics such as trauma-informed approaches, instruc- 
tional strategies, and general classroom management. 


Treatment Integrity Procedures 


Observational measures were used in both BEST in 
CLASS-E and BAU comparison conditions to assess 
teacher implementation of BEST in CLASS-E practices. 
Teacher implementation of the BEST in CLASS-E practices 
was assessed on three dimensions (adherence, competence, 
and student responsiveness) using the TIES (Sutherland 
et al., 2017), which was adapted from the BEST in CLASS 
Adherence and Competence Scale (Sutherland et al., 2014). 
Adherence refers to the extent to which the program prac- 
tices were delivered as intended, whereas competence refers 
to the level of skill and degree of responsiveness demon- 
strated by a teacher when delivering the practices (Carroll 
& Nuro, 2002). Student responsiveness refers to the degree 
to which the teachers’ attempts to engage students held the 
interest and participation of students (Dane & Schneider, 
1998; Durlak & DuPre, 2008). 


Data analyses. To test whether BEST in CLASS-E 
influenced change in student behavioral and academic 
outcomes, teacher—student relationships, and classroom 


quality, condition (scored as 0 = BAU classrooms and | = 
BEST in CLASS-E classrooms) was regressed on posttest 
student outcomes, teacher—student relationships, and class- 
room quality scores, while controlling for pretest student, 
student-teacher, and classroom scores (i.e., pretest scores 
were regressed on posttest scores). Therefore, direct effects 
are interpreted as the extent to which BEST in CLASS-E 
predicted change in student behavior, student academic 
achievement, student-teacher relationships, and classroom 
quality at posttest. 

Analyses were conducted using Stata, version 15.1 
(StataCorp, 2017). For the models predicting Problem 
Behavior, Social Skills, Reading Scores, Math Score, Conflict, 
and Closeness, a multilevel approach using the “mixed” com- 
mand was used, with students nested in teachers. These mod- 
els were estimated using restricted maximum likelihood, and 
a Kenward—Roger degrees of freedom correction (Kenward & 
Roger, 1997, 2009) was applied to account for the small sam- 
ple size. This approach is in line with recent recommendations 
for obtaining more precise estimates from multilevel models 
with a small number of clusters (McNeish & Stapleton, 2016). 

The models predicting Classroom Organization, 
Instructional Support, and Emotional Support did not con- 
verge using this approach. Therefore, we used linear regres- 
sion with cluster (teacher)-robust standard errors (Huber, 
1967; White, 1980). Using this specification accounts for the 
nonindependence of students by adjusting the standard error 
estimates and, therefore, removes variance due to the repeti- 
tion of children across teachers (Asparouhov & Muthen, 
2006). Without accounting for the nonindependence in the 
data, the estimated standard errors would be inflated result- 
ing in a greater chance of committing a Type I error. 


Results 


Missing data patterns revealed that two teachers were miss- 
ing data on posttest study variables (see Table 2). 
Independent samples ¢ tests and a one-way analysis of vari- 
ance (ANOVA) showed that teachers for whom complete 
data were not available did not differ from teachers with 
complete data on demographics (e.g., age, years of teaching 
experience, race/ethnicity, gender). In addition, Little’s 
(1988) missing completely at random test showed that data 
were missing completely at random, y7(3) = 4.13, p = .25. 
Missing data patterns also revealed that one student was 
missing pretest WJ-III scores, four students were each miss- 
ing posttest scores on the STRS and scores on the SSIS, and 
five students were each missing posttest WJ-III scores. 
Independent samples ¢ tests and a one-way ANOVA showed 
that students for whom complete data were not available did 
not differ from students with complete data on demo- 
graphics (e.g., age, race/ethnicity, gender). In addition, 
Little’s (1988) missing completely at random test showed 
that data were missing completely at random, ¥7(20) = 
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Table 2. Descriptive Statistics for Study. 


Condition Instrument/subscale Tx (n) BAU (n) Tx M (SD) BAU M (SD) = Tx Min—Max BAU Min—Max 
Pretest teacher level 
CLASS 
Classroom Organization 14 12 3.91 (0.88) 3.93 (1.15) 2.83-5.25 1.5-5.42 
Instructional Support 14 12 1.95 (0.63) 1.82 (0.56) 1.17-3.08 |.08—-2.75 
Emotional Support 14 12 4.62 (0.86) 4.24 (0.79) 3.19-6.06 2.44-5.13 
Pretest child level 
STRS 
Closeness 25 20 30.84 (5.3) 30.25 (5.5) 20-39 22-40-35 
Conflict 25 20 27.04 (6.2) 26.6 (6.2) 12-36 15-36 
SSIS 
Social Skills 25 20 71.76 (10.5) 77,7 (12.9) 49-88 56-99 
Problem Behavior 25 20 131.6 (13.5) 127.85 (14.5) 106-154 99-152 
Academic Competence 25 20 86.36 (18.1) 90.35 (11.54) 63-122 67-114 
W)-III 
Reading 26 20 25.31 (13.1) 28.19 (14.1) 4-60 12-56 
Math 26 20 13.32 (4.35) 15.85 (6.2) 4-28 6-31 
Posttest teacher level 
CLASS 
Classroom Organization 13 II 4.35 (0.85) 4.02 (1.19) 3.00-5.75 1.75-5.5 
Instructional Support 13 11 1.7 (0.51) 1.73 (0.55) 1.00-2.5 |.08-2.75 
Emotional Support 13 II 4.8 (0.81) 4.4 (1.03) 3.44-6.75 2.31-5.63 
Posttest child level 
STRS 
Closeness 25 16 35.00 (3.5) 31.12 (4.8) 28-40 23-40 
Conflict 25 16 22.72 (6.4) 24.5 (7.30 14-35 8-34 
SSIS 
Social Skills 25 16 85.32 (13.5) 86.06 (11.9) 55-122 61-109 
Problem Behavior 25 16 118.24 (15.6) 125.4 (19.9) 91-150 92-156 
Academic Competence 25 16 90.84 (17.82) 91.25 (13.85) 65-122 65-109 
W)-III 
Reading 24 16 27.42 (12.7) 32.56 (14.1) 9-59 14-59 
Math 24 16 14.96 (4.3) 18.63 (7.4) 8-27 10-32 


Note. Tx = Treatment; BAU = business as usual group; CLASS = Classroom Assessment Scoring System (Pianta et al., 2008); STRS = Student— 
Teacher Relationship Scale (Pianta, 1993); SSIS = Social Skills Improvement System—Rating Scale (Gresham & Elliott, 2008); WJ-III = Woodcock— 


Johnson—Ill Test of Achievement: Brief Battery (Woodcock et al., 2007). 


15.10, p = .77. The full information maximum likelihood 
(FIML) estimator was used to account for these missing 
data. This estimator retains the statistical power of the full 
analytic sample while minimizing bias in parameter 
estimates when data cannot be presumed to be missing 
completely at random (Enders, 2001). 

The first goal was to examine the extent to which BEST 
in CLASS-E influenced change in student behavior (i.e., 
teacher-reported student problem behavior and social skills 
on the SSIS-RS; Gresham & Elliott, 2008) compared with 
BAU. Controlling for pretest scores, results revealed com- 
pared with BAU, BEST in CLASS-E did not significantly 
influence student social skills (B = 4.82, p = .16,d = 0.21; 
see Table 3). However, results revealed, compared with 
BAU, BEST in CLASS-E significantly changed student 
problem behavior (B = —11.83, p < .05, d = —0.32). 


Next, we examined the influence of BEST in CLASS-E on 
student academic achievement compared with BAU (using 
WS-IIL; Woodcock et al., 2007). Controlling for pretest scores, 
compared with BAU classrooms, BEST in CLASS-E did not 
significantly influence student math and reading scores at post- 
test (B = —0.51 and—0.01, p = .56 and .99, d = —0.09 and 0.00, 
for math and reading scores, respectively). 

The third goal was to examine the influence of BEST in 
CLASS-E on teacher-student relationships compared with 
BAU (using the STRS; Pianta, 1993). Controlling for pretest 
scores, BEST in CLASS-E did not significantly change 
teacher-student conflict (B = —1.86, p = .23, d = —0.18) in 
comparison with BAU classrooms. However, BEST in 
CLASS-E did significantly change teacher—student closeness 
(B = 3.66, p < .001, d = 0.55) in comparison with BAU 
classrooms. 
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Table 3. Pretest Scores and Treatment Condition as 
Predictors of Posttest Outcomes. 


Pretest score Condition 
Scale B SE B SE 
Problem Behavior 0.64 0.15 -11.83* 5.48 
Social Skills 0.45" 0.24 4.82 3.31 
Reading Scores 0.93°" 0.05 -0.01 1.47 
Math Scores 0.97" 0.09 -0.51 1.04 
Conflict 055° 0.14 -1.33 1.54 
Closeness 0.26" 0.09 3.66" 1.0 
Classroom Organization 0.37" 0.18 0.29 0.40 
Instructional Support 0.12 0.17 -0.09 0.20 
Emotional Support 0.62" 0.17 0.11 0.27 
Note. Condition is scored as 0 = business as usual; | = BEST in 


CLASS-E. For the models predicting Problem Behavior, Social Skills, 
Reading Scores, Math Score, Conflict, and Closeness, a multilevel 
approach was used, with students nested in teachers. These models 
were estimated using restricted maximum likelihood, and a Kenward— 
Roger degrees of freedom correction was applied to account for 

the small sample size. The models predicting classroom organization, 
instructional support, and emotional support did not converge using this 
approach. Therefore, we used linear regression with cluster (teacher)- 
robust standard errors. 

“p< .05. “p< .01. “p < 001. 


Finally, we tested the extent to which BEST in CLASS-E 
changed the overall classroom climate compared with BAU 
(using CLASS; Pianta et al., 2008). Controlling for pretest 
scores, BEST in CLASS-E did not significantly change 
Classroom Organization (B = 0.29, p = .47, d = 0.11), 
Instructional Support (B = —0.09, p = .66, d = —0.07), or 
Emotional Support (B = 0.11, p = .69, d = 0.06) compared 
with BAU classrooms. 


Treatment Integrity 


Adherence, competence, and student responsiveness scores 
increased from pretest to posttest in BEST in CLASS-E 
classrooms, whereas decreases in these integrity dimen- 
sions were observed in BAU classrooms from pretest to 
posttest (see Table 4 for descriptive statistics). Effect size 
estimates were d = 0.67 and d = 1.70, respectively, for 
adherence and competence. 


Discussion 


The purpose of this study was to examine the effect of 
BEST in CLASS-E, a Tier 2 intervention delivered by 
teachers, on student problem behavior, academic achieve- 
ment, teacher—student relationships, and classroom quality 
using a randomized controlled trial design. Results suggest 
that the intervention had a small to moderate effect on stu- 
dent behavior and teacher—student relationships; however, 
no effects were found for academic achievement or 


classroom quality. Each of these findings will be discussed, 
followed by implications for future research and limitations 
of the current study. 

In terms of student problem behavior, a small effect was 
noted on the Problem Behavior Scale of the SSIS. These 
findings replicate the effects of BEST in CLASS on 
preschool-aged children, which found effect sizes of —0.42 
for problem behavior (Sutherland, Conroy, Algina, et al., 
2018). Results suggest the teacher training and practice- 
based coaching of BEST in CLASS-E is a promising 
approach for reducing early elementary-age students’ 
problem behavior; it should be pointed out that TIES data 
suggest that teachers in the BEST in CLASS-E condition 
had increases from pretest to posttest in adherence and 
competence of delivery of the BEST in CLASS-E practices, 
in comparison with the BAU condition, which likely 
contributed to changes in students’ behavior. This is 
important, as previous research on BEST in CLASS has 
shown that although adherence is important, teacher com- 
petence of delivery mediates treatment effects (Sutherland 
Conroy, McLeod, et al., 2018). Future work is needed to 
determine whether these mechanisms operate similarly for 
early elementary school teachers and students. 

At the same time, no effects were found on the Reading 
or Math scales of the WJ-III. Thus, although BEST in 
CLASS-E appears to have a desirable effect on student 
behavior, this effect may not be enough to result in short- 
term academic gains, as the intervention only lasts approxi- 
mately 14 weeks. Other prevention work (e.g., August et al., 
2002; Kellam et al., 2014) has found more distal effects for 
outcomes not targeted by intervention but yet associated 
with those targeted outcomes. For example, Kellam et al. 
found effects of the Good Behavior Game, a universal 
classroom management program delivered by teachers in 
first grade, on risky sexual behavior and drug abuse in 
young adulthood. It may be that the proximal effects of 
BEST in CLASS-E on student behavior contribute to more 
distal improvements in academic achievement via reduc- 
tions in classroom behavior problems and increased engage- 
ment; however, this is an empirical question that calls for 
longitudinal follow-up of students. 

Furthermore, given longitudinal associations in the lit- 
erature between early teacher—student relationships and 
interactions and students’ academic achievement (e.g., 
Pakarinen et al., 2017; Spilt, Hughes, et al., 2012), potential 
effects of BEST in CLASS-E from the current study on 
teacher-reported closeness may also potentially contribute 
to distal student outcomes. The effect size in the current 
study was 0.55, compared with a smaller effect (d = 0.26) 
noted in the BEST in CLASS preschool study (Sutherland, 
Conroy, Algina, et al., 2018). The elementary version of 
BEST in CLASS does include specific training and coach- 
ing on supportive relationships (see Sutherland et al., 2019), 
whereas the preschool version does not specifically target 
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Table 4. TIES Adherence and Competence Descriptives at Pre- and Posttest by Study Group. 


BEST in CLASS-E BAU 

Descriptive M SD Range M SD Range 
Adherence 

Pretest 2.88 1.62 1.04-5.11 2.63 1.62 1.25-5.15 

Posttest 3.18 1.66 1.50-5.54 2.14 1.44 1.06-4.56 
Competence 

Pretest 3.79 1.06 2.00-4.67 3.82 0.61 3.00-4.50 

Posttest 4.78 0.31 4.50-5.25 3.56 0.97 2.00-4.60 
Student responsiveness 

Pretest 481 4.79 

Posttest 5.08 4.45 


Note. BAU = business as usual condition; BEST in CLASS-E = treatment condition; CLASS = Classroom Assessment Scoring System. 


this as one of the instructional practices. However, given 
the small sample size in the current study, compared 
with the preschool study, the magnitude of this effect is 
surprising. On one hand, perhaps BEST in CLASS-E has a 
greater impact on the teacher-reported closeness with older 
children via improvements in teacher—student interactions 
during increasingly demanding academic interactions. 
However, we surmise that an additional explanation may be 
the overall level of risk presented in the classrooms as mea- 
sured by the PPCAS measure (Abry et al., 2018; National 
Center for Education Statistics, 1993). Specifically, teach- 
ers completed this measure as an estimate of the overall 
level of classroom risk (e.g., toxic stress, learning prob- 
lems) faced by students in their classrooms. Ratings on this 
measure (M = 3.00 and 3.01) were higher than previous 
studies using this measure (M = 1.78; Abry et al., 2018), 
which is not surprising given the environmental challenges 
present in the community where these schools were located 
(i.e., high poverty, violence exposure, food desert, limited 
transportation). It may be that given the overall needs of a 
large number of students in these classrooms that the teacher 
attempts at increasing positive interactions and being 
supported to do so by the BEST in CLASS-E training and 
coaching led to teachers feeling closer to focal students. 

At the same time, there were no significant effects noted 
for teacher-reported conflict or any of the dimensions of the 
CLASS measures, which assesses overall classroom quality. 
Previous research on the preschool version of BEST in 
CLASS (Conroy, Sutherland et al., 2019) found small effects 
for reductions in teacher-reported conflict (d = —0.29, com- 
pared with d = —0.18 in the current study) and small to mod- 
erate effects for the three dimensions of the CLASS. It is 
possible the small sample size may have contributed to an 
increased likelihood of Type II error due to low statistical 
power. Alternatively, BEST in CLASS-E may not have been 
strong enough to overcome the existing risk as assessed by 
the PPCAS. Thus, although teachers reported feeling closer 


to the focal students as a result of the training and coaching 
support provided by BEST in CLASS-E, these supports did 
not result in their feeling significantly less conflict toward 
the focal students nor were these supports sufficient to over- 
come high rates of classroom adversity. 


Study Limitations and Implications 


Several limitations of this study should be considered when 
interpreting results. First, problem behavior and social skills 
in the current study were assessed via the SSIS, which is a 
teacher-report measure. Future work should also include 
direct observations of student behavior to strengthen study 
findings. Furthermore, this study, which was an underpow- 
ered promise study of an Institute of Education Sciences 
development project, utilized a small sample of teachers 
and students from a geographically unique setting; there- 
fore, generalizing these findings to other students, teachers, 
or communities should be done cautiously. The small sam- 
ple may limit our ability to detect group differences that 
may in fact be present. Future research should use samples 
large enough to detect group differences across diverse set- 
tings to advance our understanding of BEST in CLASS-E 
effects on elementary students with problem behavior who 
are at risk of EBD and the classrooms in which they are 
taught. Relatedly, the data from the PPCAS suggest that 
these classrooms served students who had experienced a 
number of adverse events. Thus, results from this study may 
not generalize to classrooms or schools where families and 
students experience fewer adverse events. Also, the lack of 
Tier | interventions in place in the schools in the current 
study limit our ability to assess the fit of BEST in CLASS-E 
within a PBIS framework delivered with high fidelity, and 
future work should attempt to implement BEST in CLASS-E 
within PBIS systems delivered with high fidelity. In addi- 
tion, although the treatment integrity data from the current 
study replicate findings from the BEST in CLASS 
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preschool study, the small sample size in the current study 
precludes examining adherence and competence as poten- 
tial mediators of treatment effects. Future research with 
larger samples can help examine the relationship between 
outcomes and treatment integrity. In addition, the low ICC 
noted for the Instructional Support domain of the CLASS 
limits the interpretability of this measure. Finally, although 
there were no effects found for academic outcomes in the 
current sample, longitudinal research is necessary to help 
examine potential distal effects of interventions such as 
BEST in CLASS-E on downstream outcomes (e.g., Kellam 
et al., 2014). 


Conclusion 


BEST in CLASS-E appears to have promise for improving 
behavioral outcomes and teacher—student relationships for 
young elementary students with and at risk of EBD. These 
findings replicate those found for BEST in CLASS-E in 
early childhood classrooms and highlight the promise of the 
professional development approach—evidence-based prac- 
tice elements supplemented with practice-based coaching— 
used in BEST in CLASS-E. At the same time, more work is 
needed to better understand potential long-term impacts on 
both teacher and student behavior, not to mention whether 
there are distal effects on academic achievement. Better 
understanding these downstream effects can help inform 
future work on Tier 2 interventions such as BEST in 
CLASS-E to maximize their impact on outcomes for this 
group of vulnerable students. 
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